Skip to content

fix(eval): agent cost was the wrong model's price, and the budget was the wrong unit - #545

Merged
sunholo-voight-kampff merged 3 commits into
devfrom
fix/agent-eval-cost-provenance-and-work-gates
Aug 7, 2026
Merged

fix(eval): agent cost was the wrong model's price, and the budget was the wrong unit#545
sunholo-voight-kampff merged 3 commits into
devfrom
fix/agent-eval-cost-provenance-and-work-gates

Conversation

@sunholo-voight-kampff

Copy link
Copy Markdown
Collaborator

Found while confirming OpenAI's 2026-07-30 GPT-5.6 price cut. Three defects, one new tally.

1. Agent cost billed the wrong model

Executors that compute cost from tokens used their own hardcoded table, so the codex harness billed every model it ran at gpt-5-codex's $1.25/$10 per 1M. gpt5-6-luna and gpt5-4-mini banked at identical rates despite really costing $0.20/$1.20 and $0.75/$4.50.

Task.Pricing now carries per-model rates; ResolveCostModel prefers them and falls back to the executor's table. A present-but-zero rate is honoured, so free local models aren't billed at cloud prices.

The cost-kill path was already per-model correct — which meant banked cost and kill threshold came from two different price tables:

luna/graph_bfs (256,987 in / 2,136 out)
banked cost_usd $0.34259 (codex rates) — above the $0.30 cap
budget saw $0.26980 (luna rates) — under the cap
outcome PASS

2. cost_usd never said whether anyone was charged

codex (auth_mode: chatgpt) and claude (OAuth) run on subscriptions, so their dollars are list-price equivalents — yet they summed with metered OpenRouter/Vertex spend under one label, directly beneath a v1.0 KPI whose numerator is attributable metered dollars.

Each executor now classifies its own auth lane. Banked as cost_provenance, stored on chain stages (schema v17), and split by ClassifyStageCost into a subscription status that CostRollup keeps out of TotalKnownCost. Absent labels read as unknown, never metered — nothing is backfilled by guess.

3. The budget was denominated in the wrong unit

A dollar ceiling buys tokens in inverse proportion to price, so nominally-uniform $0.30/$0.50 caps spanned 24× in real work:

model before after
opencode-or-deepseek-v4-flash 3.40M tok 3.00M
claude-sonnet-4-6 (anchor) 0.14M 3.00M
claude-haiku-4-5 0.25M 3.00M
gpt5-6-luna 1.20M 3.00M

claude-sonnet-4-6 — the longitudinal anchor everything else is scored against — was the most starved model in the suite. It had no budgets: block at all and fell through to a formula that buys almost nothing at $3/$15. A suite asking "does the agent loop rescue weak models" was giving the weakest the most rope.

budgets.max_tokens_per_bench sets one shared 3.0M ceiling. Subscription lanes gate on tokens alone and now sit identical; their dollar caps were raised only so the token gate binds — not a spend increase, those lanes aren't billed. Metered opencode lanes keep their dollar ceiling as a real spend control with tokens as a second bound.

Task.MaxTokensPerBench had been plumbed to executor.Task since M-EVAL-OS-LONGITUDINAL, but only opencode ever honoured it. codex and claude now enforce it too.

4. End-of-run cost tally

eval-suite now prints per-model cost then totals split by provenance. METERED (actually billed) is the only line that answers "what did this cost us".

Verification

Build, go vet, make check-boundaries, and tests across executor, eval_harness, observatory, eval_analysis, coordinator, server and storage. Tally rendered against the real v0.30.0 baseline — 2,090 rows, all correctly classified unknown (they predate the label).

Reviewer notes

  • ⚠️ The claude dollar caps ($10.80, $3.60) are free on this rig's OAuth but become real per-benchmark exposure under AILANG_AUTH_MODE=apikey in cloud dispatch. Flagged inline at both sites.
  • Metered lanes still span 15.5× in work — now a stated policy choice, not an accident. Equalizing would cost ~$260/run on glm-5-2.
  • Existing v0.30.0 rows keep their wrong cost_usd; correcting them would mean fabricating numbers, so the annotation route was taken (same call as CAVEATS.md).

🤖 Generated with Claude Code

sunholo-voight-kampff and others added 2 commits July 31, 2026 12:22
… the wrong unit

Two defects found auditing OpenAI's 2026-07-30 GPT-5.6 price cut, plus the
end-of-run tally that would have surfaced either one sooner.

1. PER-HARNESS PRICING. Executors that compute cost from token counts used
   their own hardcoded table, so the codex harness billed EVERY model it ran
   at gpt-5-codex's $1.25/$10 per 1M — gpt5-6-luna and gpt5-4-mini banked at
   identical rates despite really costing $0.20/$1.20 and $0.75/$4.50.
   Task.Pricing now carries the per-model rates and ResolveCostModel prefers
   them; a present-but-zero rate is honoured so free local models are not
   billed at cloud prices. The cost-KILL path was already per-model correct,
   which meant banked cost and kill threshold came from two different price
   tables: luna/graph_bfs banked $0.34259 at codex rates while the $0.30
   budget that spared it saw $0.26980 at luna's.

2. COST PROVENANCE. cost_usd said nothing about whether anyone was charged.
   codex (auth_mode chatgpt) and claude (OAuth) run on subscriptions, so
   their dollars are list-price equivalents, not spend — yet they summed
   with metered OpenRouter/Vertex spend under one label, directly beneath a
   v1.0 KPI whose numerator is attributable METERED dollars. Each executor
   now classifies its own auth lane; banked as cost_provenance, stored on
   chain stages (schema v17), split out by ClassifyStageCost into a new
   `subscription` status that CostRollup keeps OUT of TotalKnownCost.
   Absent labels read as unknown, never metered — nothing is backfilled.

3. WORK GATES. A dollar ceiling buys tokens in inverse proportion to price,
   so nominally-uniform $0.30/$0.50 caps spanned 24x in real work:
   claude-sonnet-4-6, the longitudinal ANCHOR, got 0.14M tokens against
   deepseek-v4-flash's 3.40M — it had no budgets block at all and fell
   through to a formula that buys almost nothing at $3/$15. A suite asking
   "does the agent loop rescue weak models" gave the weakest the most rope.
   budgets.max_tokens_per_bench sets one shared 3.0M ceiling; subscription
   lanes gate on tokens alone and now sit identical. Task.MaxTokensPerBench
   had been plumbed since M-EVAL-OS-LONGITUDINAL but ONLY opencode honoured
   it; codex and claude now enforce it too.

Metered opencode lanes keep their dollar ceiling as a real spend control with
tokens as a second bound — still cost-limited, but for a stated reason.

eval-suite now prints a cost tally at the end, split by provenance, so the
metered figure is never conflated with subscription arithmetic again.

Verified: build, go vet, make check-boundaries, and tests across executor,
eval_harness, observatory, eval_analysis, coordinator, server and storage.
Tally rendered against the real v0.30.0 baseline (2,090 rows, all correctly
classified unknown — they predate the label).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The provenance and token-gate work pushed both executors past the 800-line
check-file-sizes gate (codex 817, claude 843) — a CI failure, not a warning.

Extracted into a cost.go per package: authLane() + CostModel() for both, plus
getModel() for claude. These belong together — what a run costs and whether
anyone was charged for it are the same question asked twice.

codex 817 -> 761, claude 843 -> 778. No behaviour change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sunholo-voight-kampff

Copy link
Copy Markdown
Collaborator Author

Mission-loop triage (iteration 150). This PR is an orphan: it has zero mentions in the V1 mission charter and zero in the mission log (control: #544 has 4 log mentions, so the search sees positives). It was opened 2026-07-31, last touched 11:04 the same day, and nothing has referenced it since — so it was never picked up by an iteration and no one is waiting on it.

Its purpose is still live — I checked rather than assuming. The standing rule here is that an OPEN + long-untouched PR is evidence toward superseded, so I measured whether dev had solved this independently. It has not:

symbol / file hits at dev
ResolveCostModel 0
CostProvenance / cost_provenance 0
internal/eval_harness/cost_tally.go absent
internal/executor/codex/cost.go absent

Control: internal/executor/cost.go (a file this PR only modifies) does exist, so the absences are measurements rather than a mistyped path.

So the two defects it fixes are still present: agent cost banked from the executor's hardcoded table rather than per-model rates, and cost_usd summing subscription list-price-equivalents together with genuinely metered spend under one label.

State: MERGEABLE: CONFLICTING, and the branch is 125 commits behind dev (2 commits ahead). It cannot be merged as-is, and the surface it touches (47 files across eval_harness, executor/*, observatory, storage/firestore) has moved a lot in those 125 commits — notably the v0.32.0 confidence-gating work and the v0.33.0 recorded-stream changes.

Not actioned this iteration — a triage sweep never outranks the queue head by itself, and this is a rebase-and-revalidate job, not a merge. Filed as a queue row in the charter so it stops being invisible to the loop.

🤖 Generated with Claude Code

sunholo-voight-kampff added a commit that referenced this pull request Aug 6, 2026
… scope decision after 2 blocked quorum rounds

Pick: the queue head, m-net-effect-proxy-boundary (D5 Option B). NEW-DOC + quorum.
Designer codex gpt-5.6-sol (rotation advanced from claude:claude-fable-5).

Outcome: 662-line design doc with 19 verification rows and a silent-revert check on
every acceptance criterion. Quorum BLOCKED twice; item parked needs-human-review on
one decision (D-6). Planner/executor/evaluator not fired.

R1 caught a genuine design defect (gemini-3-1-pro): target-IP resolution specified in
TWO places — the existing preflight resolveAndValidateIP and the new RoundTripper — a
TOCTOU DNS-rebinding race plus a broken-proxied-request risk on hosts without external
DNS. Routed without a controller-invented resolution; the designer made the direct
RoundTripper the sole resolve-validate-dial site and skips local target DNS entirely on
proxy routes.

R1's other objection was an unverified premise, so the controller measured it rather
than forwarding it: production has 0 custom RoundTrippers, 0 DefaultTransport uses,
0 Transport.Clone, no shared HTTP factory (control: 29 inline http.Client{} sites).
That pass also produced the fact the doc lacked — Go's DefaultTransport sets
Proxy: ProxyFromEnvironment (transport.go:46-48, go1.26.5) — so bare clients are
already inside the egress boundary and only hand-built nil-Proxy transports can
escape. That turns "we found seven sites" into "seven is all there can be".

R2's surviving objection asks that the completeness gate become a go/packages AST
analyzer. Not carve-out-eligible (expands scope, needs judgment), so Standing rule 2
applies: park, do not force through. The decision was made cheap to answer by testing
the reviewer's own hypothesis — all five constructions the analyzer would catch are
ZERO at HEAD, each with a firing control. Two of those five controls failed on first
run and were re-run before any number was used.

Also: corrected the stale sweep-batch row (#588 was already closed 2026-08-05; and M2
gated the live-network SUBTEST at net_test.go:364, not the whole TestNetHttpPost
function). Filed orphaned PR #545 as a queue row — found by iteration 149's new
died-mid-flight check on its first independent use; 125 commits behind, zero charter
and zero log mentions, and its purpose is NOT superseded (measured at HEAD).

Retro — one skill edit (>=2-instance bar, purely additive: 38 insertions, 0 deletions):
Gate 2 gains rule 3f — a reviewer's objection is a claim too, and on an
"unverified premise" objection the controller's job is to MEASURE it, because the
measurement can refute the objection outright or shrink the work. Instance 1:
iteration 126. Instance 2: this iteration.

Metered: $0.1785 of the $5 ceiling (quorum x2 only; codex is OAuth-subscription).

Co-Authored-By: codex <gpt-5.6-sol>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR #545 sat open since 2026-07-31, 154 commits behind dev and CONFLICTING.
The charter carried it as a "47 files ... rebase-and-revalidate job, not a
merge". Measured rather than inherited: the merge produces exactly THREE
conflicted files, one of them a changelog.

Resolutions, each from reading both sides:

- internal/eval_harness/agent_runner_multi.go — UNION. dev added a fallback
  that computes cost from tokens when the executor reports zero; this branch
  adds CostProvenance. They are complementary, and the fallback does NOT
  reintroduce the two-price-tables defect this PR exists to fix:
  CalculateCostWithBreakdown resolves through GlobalModelsConfig.Models[key],
  the same models.yml per-model rates the branch installs as task.Pricing.

- cmd/ailang/eval_benchmark.go — took dev's side. dev extracted the agent
  path into cmd/ailang/eval_benchmark_agent.go, so the 427-line conflict is
  a code MOVE, not a rewrite. This branch's real delta to that block is four
  lines; two were re-applied verbatim into the extracted file (the
  CostProvenance field and the UpdateStageMetrics argument) and the third
  auto-merged in place.

- changelogs/v0.18-current.md — union, [Unreleased] above [v0.33.1]. Zero
  content overlap between the two sides (control: 6 "Added" headings on
  dev's side, so the zero is a measurement).

Validation, all outside any sandbox:
- make test: rc=0, 107 packages ok, 0 FAIL (control: 7070 --- PASS lines,
  so the FAIL matcher would have fired).
- go build ./...: the only failure is cmd/wasm, which fails IDENTICALLY on
  pristine dev — baselined rather than assumed. Zero new build failures.
- make check-changelog / check-file-sizes / check-boundaries / fmt-check /
  vet: all rc=0.
- models.yml lost nothing: 111 model keys on both sides, and every dev key
  is present in the merged file (comm -23 empty), including the
  pi-or-deepseek-v4-flash lane added after this branch was cut.
- Migration numbering is clean: dev is at v16, this adds v17.
- go.sum and go.mod are byte-identical before and after the run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sunholo-voight-kampff added a commit that referenced this pull request Aug 7, 2026
…een, and

the charter's own scale warning was a claim nobody had measured

PICK: #545, because both items above it are parked on a human (#604 on D-2,
#613 on D-1) and iteration 157's own Next names it as the fallback.

HEADLINE: the charter has carried #545 since iteration 149 behind a warning
reading "47 files ... a surface that moved a lot ... a rebase-and-revalidate
job, NOT a merge". Every number in it is true and the conclusion is false.
The merge produces THREE conflicted files, one hunk each: a changelog union
with zero content overlap, a two-line struct-literal union, and one that looks
enormous (427 lines ours vs 1 theirs) and is a code MOVE — dev extracted the
agent path into cmd/ailang/eval_benchmark_agent.go, and #545's real delta to
that block is four lines. Three iterations deferred the item on that sentence.
A "surface moved a lot" warning measures how much OTHER work landed, never how
much of it collides.

Also recorded: the union resolution justified rather than assumed (dev's new
zero-cost fallback resolves through the same models.yml key as #545's
task.Pricing, so it does not reintroduce the two-price-table defect); the
changelog claims a v0.33.1 release for which NO TAG EXISTS (caught in my own
dashboard draft before publishing); Sonar at six consecutive analysed reds,
measured on HEAD; and the Gate-0 weekly sweep, which found 7 zero-mention
issues, closed #598 as a verified duplicate of #602, and batched the other six
into one queue row positioned below every existing pick.

STATUS rotation done with the arithmetic asserted before writing
(before=after=1864, moved=1) and a post-edit queue-row grep, per the rules
iteration 127's mass-deletion bought.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@sunholo-voight-kampff

Copy link
Copy Markdown
Collaborator Author

Unblocked by mission-control iteration 158 — CONFLICTINGMERGEABLE.

The charter has deferred this PR for three iterations behind a warning that read: "47 files across eval_harness, executor/*, observatory, storage/firestore — a surface that moved a lot in those 125 commits, so this is a rebase-and-revalidate job, not a merge." Every number in it is true. The conclusion was not, and nobody had run the merge to check.

Measured: 3 conflicted files out of 47, one hunk each.

file resolution
changelogs/v0.18-current.md union, [Unreleased] above [v0.33.1]. Zero content overlap between the sides (control: 6 Added headings on dev's side, so the zero is a measurement)
internal/eval_harness/agent_runner_multi.go union — two lines
cmd/ailang/eval_benchmark.go took dev's side. Looks enormous (427 lines ours vs 1 theirs) and is a code MOVE: dev extracted the agent path into cmd/ailang/eval_benchmark_agent.go. This branch's real delta to that block, against the merge base, is four lines — two re-applied verbatim into the extracted file, one auto-merged in place

The union in agent_runner_multi.go was justified, not just applied. dev independently added a fallback computing cost from tokens when the executor reports zero — prima facie the very two-price-tables defect this PR exists to kill. Traced rather than assumed: dev's CalculateCostWithBreakdownGlobalModelsConfig.CalculateCostForModelModels[lookupKey], and this branch's task.Pricing is built from cfg.Pricing where cfg is Models[lookupKey]. Same models.yml rates, same key — complementary, so the PR's thesis survives the merge.

Validation, all outside any sandbox, all baselined:

  • make testrc=0, 107 packages, 0 FAIL (control: 7070 --- PASS lines, so the FAIL matcher would have fired)
  • go build ./... — the only failure is cmd/wasm, which fails identically on pristine dev. Zero new build failures
  • make check-changelog / check-file-sizes / check-boundaries / fmt-check / vet — all rc=0 (gate list derived from ci.yml, not recalled)
  • models.yml lost nothing: 111 model keys on both sides, comm -23 empty, including the pi-or-deepseek-v4-flash lane added after this branch was cut
  • migration numbering clean — dev is at v16, this adds v17
  • go.sum/go.mod byte-identical before and after

Both defects this PR documents were re-verified as still live on dev before any work: ResolveCostModel 0 hits, CostProvenance 0, cost_tally.go and executor/codex/cost.go absent (control: internal/executor/cost.go present, 644 funcs in eval_harness).

Ready to merge on its own CI green.

🤖 Generated with Claude Code

@sonarqubecloud

sonarqubecloud Bot commented Aug 7, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
63.4% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@sunholo-voight-kampff
sunholo-voight-kampff merged commit 32583be into dev Aug 7, 2026
19 of 20 checks passed
@sunholo-voight-kampff
sunholo-voight-kampff deleted the fix/agent-eval-cost-provenance-and-work-gates branch August 7, 2026 08:31
sunholo-voight-kampff added a commit that referenced this pull request Aug 7, 2026
…d I was

wrong that Sonar's red was purely inherited

#545 merged as squash 32583be. Both gates green with completeness asserted:
PR run present==4/4 all success, all four REQUIRED contexts pass; post-merge
dev run present==3/3 all success. Queue row flipped to [LANDED] only on the
observed green.

Evaluator (sonnet, independent of the opus controller that resolved the merge):
PASS 77/100, zero blocking. Both substantive findings reproduced first-party
before acting. The cost-fallback/provenance union is correct for a reason the
judge did not reach: ResolveCostProvenance is TOTAL, so provenance is a
property of the auth lane rather than of who computed the number. Its coverage
finding is real and is now #615, along with pi's hardcoded AuthLaneBilled.

Refuted one of the judge's corrections by measurement: its 108 packages vs my
107 is scope, not error — make test excludes /scripts, of which exactly one
package has tests.

THE CORRECTION THAT MATTERS: earlier in this same iteration I recorded Sonar's
red as "a standing condition inherited from earlier Go work, not a regression
from this push". True of iterations 155-157, NOT true of this one. #545's head
read 63.4% coverage on new code and the merged commit reads 73.7%, down from
78.7%. It never gated (non-required), so the merge decision stands, but
"inherited" was an assumption carried across a push that changed the input —
rule 3d exactly. Likewise the consecutive count: I wrote 7 by extrapolation and
enumeration says 8.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant